Spatial cluster to identify spatial domains¶
This tutorial demonstrates how to cluster spatial transcriptomics (ST) data based on both location and gene expression information
Center for Quantitative Biology
Academy for Advanced Interdisciplinary Studies, Peking University
Lab website: http://cqb.pku.edu.cn/zenglab/
Written by Yanping Zhao
Background¶
An essential initial step in ST research is to cluster the spots and define spatially coherent regions in terms of expression data and location adjacency.
This process essentially entails classical unsupervised clustering of spots into groups according to the similarity of their gene expression profiles and spatial locations, subsequently assigning labels to each cluster.
Here, we mainly introduce the cellcharter tool
Import packages¶
In [1]:
# Import core packages
import scanpy as sc
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
# Disable annoying warnings
import warnings
warnings.filterwarnings("ignore", category=FutureWarning)
warnings.filterwarnings("ignore", category=UserWarning)
# set working directory
import os
# Direct show figure in jupyter
%matplotlib inline
%config InlineBackend.figure_format = "retina"
Import data and preprocess¶
ST data: Visium spatial transcriptomics dataset of the human lymphnode, which is publicly available from the 10x genomics website
In [ ]:
# solution1 - Import
adata = sc.read_visium('/lustre1/share/spatial_data/course/')
In [ ]:
# solution2 - Download datasets
adata = sc.datasets.visium_sge(sample_id="V1_Human_Lymph_Node")
In [3]:
# Standard processing
adata.var_names_make_unique()
adata.var["mt"] = adata.var_names.str.startswith("MT-")
sc.pp.calculate_qc_metrics(adata, qc_vars=["mt"], inplace=True)
sc.pp.filter_genes(adata, min_counts=3)
sc.pp.filter_cells(adata, min_counts=3)
sc.pp.normalize_total(adata, target_sum=1e6)
sc.pp.log1p(adata)
Dimension reduction
In [4]:
sc.pp.pca(adata)
plt.rcParams["figure.figsize"] = (3, 3)
sc.pl.pca(
adata,
color=["total_counts", "n_genes_by_counts", "pct_counts_mt", "array_row"],
dimensions=[(0, 1), (2, 3), (0, 1), (2, 3)],
ncols=2,
size=2,
)
Visualization
In [5]:
plt.rcParams["figure.figsize"] = (4, 4)
sc.pl.spatial(adata, img_key="hires", color=["total_counts", "n_genes_by_counts"])
using leiden algorithm to cluster only based on location proximity¶
In [6]:
# https://scanpy.readthedocs.io/en/stable/api/generated/scanpy.pp.neighbors.html#scanpy.pp.neighbors
# !!!!
# Based on sc.pp.neighbors function to calculate the spots proximity
# 【note】 This cluster only based on spatial location
sc.pp.neighbors(adata, n_pcs=2, n_neighbors=8, method='gauss', use_rep='spatial', metric='euclidean',
key_added='loc' # the following sc.tl.leiden function will cluster ST based on this key
)
In [7]:
adata
Out[7]:
AnnData object with n_obs × n_vars = 4035 × 22429
obs: 'in_tissue', 'array_row', 'array_col', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'n_counts'
var: 'gene_ids', 'feature_types', 'genome', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'n_counts'
uns: 'spatial', 'log1p', 'pca', 'loc'
obsm: 'spatial', 'X_pca'
varm: 'PCs'
obsp: 'loc_distances', 'loc_connectivities'
In [8]:
sc.tl.leiden(adata, neighbors_key='loc',resolution=0.3 )
In [9]:
plt.rcParams["figure.figsize"] = (4, 4)
sc.pl.spatial(adata, color=["leiden"])
using Cellcharter tool to cluster based on both transcriptome and location proximity¶
Cellcharter is the fast and GPU-based tool to handle ST data, but we only use the CPU version considering the process of installation
Import cellcharter
In [10]:
import cellcharter as cc
/home/zhaoyp/miniconda3/envs/st_cluster/lib/python3.9/site-packages/numba/core/decorators.py:246: RuntimeWarning: nopython is set for njit and is ignored
warnings.warn('nopython is set for njit and is ignored', RuntimeWarning)
/home/zhaoyp/miniconda3/envs/st_cluster/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
In [11]:
# similar with above codes
# to get the adjacency matrices
sc.pp.neighbors(adata, n_pcs=2, n_neighbors=8, method='gauss', use_rep='spatial', metric='euclidean', key_added='loc')
In [12]:
cc.gr.aggregate_neighbors(adata, n_layers=2, use_rep='X_pca', out_key='X_cellcharter', connectivity_key='loc')
100%|██████████| 3/3 [00:00<00:00, 253.40it/s]
Choose the best K¶
In [13]:
autok = cc.tl.ClusterAutoK(
n_clusters=(2,12),
max_runs=3,
convergence_tol=0.001
)
In [17]:
autok.fit(adata, use_rep='X_cellcharter')
Iteration 1/3
0%| | 0/13 [00:00<?, ?it/s]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 8%|▊ | 1/13 [00:00<00:08, 1.36it/s]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=3` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 15%|█▌ | 2/13 [00:01<00:08, 1.31it/s]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=5` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 23%|██▎ | 3/13 [00:03<00:11, 1.14s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=7` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-06. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=7` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-05. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=7` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 31%|███ | 4/13 [00:06<00:19, 2.15s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=9` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 38%|███▊ | 5/13 [00:08<00:15, 1.98s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=11` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-06. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=11` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-05. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=11` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 46%|████▌ | 6/13 [00:11<00:15, 2.28s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=13` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-06. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=13` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-05. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=13` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 54%|█████▍ | 7/13 [00:14<00:16, 2.71s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=15` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-06. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=15` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-05. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=15` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 62%|██████▏ | 8/13 [00:18<00:14, 2.96s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=17` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-06. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=17` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-05. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=17` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 69%|██████▉ | 9/13 [00:21<00:12, 3.12s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=19` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 77%|███████▋ | 10/13 [00:22<00:07, 2.36s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=21` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-06. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=21` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-05. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=21` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 85%|████████▍ | 11/13 [00:26<00:05, 2.72s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=23` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-06. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=23` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-05. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=23` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 92%|█████████▏| 12/13 [00:28<00:02, 2.69s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=25` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-06. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=25` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-05. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=25` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 100%|██████████| 13/13 [00:32<00:00, 2.49s/it]
Iteration 2/3
0%| | 0/13 [00:00<?, ?it/s]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 8%|▊ | 1/13 [00:01<00:18, 1.52s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=3` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 15%|█▌ | 2/13 [00:02<00:12, 1.16s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=5` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 23%|██▎ | 3/13 [00:03<00:10, 1.02s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=7` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 31%|███ | 4/13 [00:04<00:08, 1.08it/s]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=9` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 38%|███▊ | 5/13 [00:05<00:09, 1.17s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=11` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-06. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=11` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-05. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=11` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 46%|████▌ | 6/13 [00:08<00:12, 1.73s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=13` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-06. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=13` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 54%|█████▍ | 7/13 [00:10<00:11, 1.86s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=15` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-06. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=15` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-05. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=15` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 62%|██████▏ | 8/13 [00:15<00:13, 2.73s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=17` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-06. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=17` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-05. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=17` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 69%|██████▉ | 9/13 [00:17<00:10, 2.72s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=19` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-06. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=19` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-05. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=19` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 77%|███████▋ | 10/13 [00:22<00:09, 3.27s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=21` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-06. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=21` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-05. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=21` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 85%|████████▍ | 11/13 [00:26<00:07, 3.63s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=23` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-06. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=23` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-05. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=23` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 92%|█████████▏| 12/13 [00:30<00:03, 3.61s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=25` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-06. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=25` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-05. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=25` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 100%|██████████| 13/13 [00:34<00:00, 2.62s/it]
Iteration 3/3
0%| | 0/13 [00:00<?, ?it/s]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 8%|▊ | 1/13 [00:01<00:18, 1.56s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=3` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 15%|█▌ | 2/13 [00:03<00:17, 1.59s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=5` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 23%|██▎ | 3/13 [00:04<00:15, 1.59s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=7` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 31%|███ | 4/13 [00:05<00:11, 1.25s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=9` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 38%|███▊ | 5/13 [00:06<00:08, 1.04s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=11` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-06. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=11` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-05. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=11` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 46%|████▌ | 6/13 [00:09<00:12, 1.79s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=13` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 54%|█████▍ | 7/13 [00:10<00:08, 1.45s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=15` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-06. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=15` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-05. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=15` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 62%|██████▏ | 8/13 [00:13<00:10, 2.06s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=17` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-06. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=17` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 69%|██████▉ | 9/13 [00:15<00:08, 2.05s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=19` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-06. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=19` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-05. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=19` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 77%|███████▋ | 10/13 [00:19<00:07, 2.54s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=21` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-06. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=21` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-05. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=21` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 85%|████████▍ | 11/13 [00:22<00:05, 2.90s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=23` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-06. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=23` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-05. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=23` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 92%|█████████▏| 12/13 [00:26<00:03, 3.22s/it]GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=25` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-06. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=25` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-05. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=25` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. 100%|██████████| 13/13 [00:30<00:00, 2.37s/it]
In [18]:
cc.pl.autok_stability(autok)
In [19]:
adata.obs['cluster_cellcharter'] = autok.predict(adata, use_rep='X_cellcharter')
GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers.
In [20]:
sc.pl.spatial(
adata,
color=['cluster_cellcharter'],
img=None,
size=1.5
)
define k = 10¶
In [21]:
gmm = cc.tl.Cluster(
n_clusters=10,
random_state=12345
)
In [22]:
gmm.fit(adata, use_rep='X_cellcharter')
adata.obs['cluster_cellcharter_10'] = gmm.predict(adata, use_rep='X_cellcharter')
GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=19` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-06. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=19` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. Cholesky decomposition failed. Retrying with covariance regularization 9.999999999999999e-05. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=19` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=300` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers. `Trainer.fit` stopped: `max_epochs=1` reached. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers.
GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] SLURM auto-requeueing enabled. Setting signal handlers.
In [23]:
sc.pl.spatial(
adata,
color=['cluster_cellcharter_10'],
img_key="hires",
size=1.5
)
Run pipeline on server¶
(1) load conda environment
In [ ]:
source /lustre1/share/miniconda3/bin/activate
conda activate cluster
(2) create a personal output directory and run the pipeline
In [ ]:
usrname=xhl # replace with your own user name
mkdir /lustre1/share/spatial_cluster/output/${usrname}
python /lustre1/share/spatial_cluster/spatial_cluster.py --path /lustre1/share/data/course/data/V1_Human_Lymph_Node --output /lustre1/share/spatial_cluster/output/${usrname}
cd /lustre1/share/spatial_cluster/output/${usrname}
ls
(3)copying data to local
usrname=C
scp -i {A} -P 16222 -r {B}@162.105.160.16:/lustre1/share/spatial_cluster/output/${usrname} {D}
-i {A} SSH private key for authentication
-P 16222 Remote SSH port (not default 22)
-r Recursively copy a directory
{B}@162.105.160.16 Remote username and server IP
/lustre1/share/spatial_cluster/output/${usrname} Remote directory to download
{D} Local destination folder
In [ ]:
usrname=xhl
scp -i yy.txt -P 16222 -r bjx164_pkuhpc@162.105.160.16:/lustre1/share/spatial_cluster/output/${usrname} /Users/xuhanlin/Desktop/2025pre/1208CTI
Reference¶
Hu, Y., Xie, M., Li, Y. et al. Benchmarking clustering, alignment, and integration methods for spatial transcriptomics. Genome Biol 25, 212 (2024). https://doi.org/10.1186/s13059-024-03361-0
Varrone, M., Tavernari, D., Santamaria-Martínez, A. et al. CellCharter reveals spatial cell niches associated with tissue remodeling and cell plasticity. Nat Genet 56, 74–84 (2024). https://doi.org/10.1038/s41588-023-01588-4